Chapter 3 - Linear Regression

This chapter is about linear regression, a very simple approach for supervised learning. Linear regression is a useful and widely used statistical learning method.

We use linear regression for predicting a _____________ response.

Simple Linear Regression

A very straightforward approach for predicting a _____________ response $Y$ on the basis of a _____________.

It assumes that there is approximately a linear relationship between $X$ and $Y$ . Mathematically, we can write this linear relationship as

We will sometimes describe this relationship by saying that we are regressing $Y$ on $X$ (or $Y$ onto $X$).

Example 1:

Recall the Advertising data from Chapter 2. In this data, sales (in thousands of units) for a particular product and advertising budgets (in thousands of dollars) for TV, radio, and newspaper media are recorded.

AdvertisingData <- read.csv("https://raw.githubusercontent.com/nguyen-toan/ISLR/master/dataset/Advertising.csv", header = TRUE, sep = ",")
AdvertisingData <- AdvertisingData[,-1]
head(AdvertisingData)

##      TV Radio Newspaper Sales
## 1 230.1  37.8      69.2  22.1
## 2  44.5  39.3      45.1  10.4
## 3  17.2  45.9      69.3   9.3
## 4 151.5  41.3      58.5  18.5
## 5 180.8  10.8      58.4  12.9
## 6   8.7  48.9      75.0   7.2

Let $X$ represents TV advertising and $Y$ represent sales. Write the model for regressing sales onto TV:

____ and _____ are two unknown constants that represent the ____ and ____ terms in the linear model.
Together, ____ and ____ are known as the model ____ or ____.
Once we have used our training data to produce estimates (how?) ____ and ____ for the model coefficients, we can predict future sales on the basis of a particular value of TV advertising by computing

where ____indicates a prediction of $Y$ on the basis of $X = x$. Here we use a hat symbol, to denote the estimated value for an unknown parameter or coefficient, or to denote the predicted value of the response.

Estimating the Coefficients

Note: These scatter plots here are NOT the actual ones from Advertising data.

What is the line of best fit? _____________________

$y =$
$\hat{y} =$
$y - \hat{y} =$

We get the LS line (estimates for the model coefficients) by minimizing the sum of squares of residuals (RSS)

\[RSS = \] Note: How do we minimize RSS? Derivatives are used to minimize RSS (outside the scope of our class).

After minimizing RSS we get our estimates ____ and ____ for the model coefficients:

The standard errors associated with $\hat{\beta_0}$ and $\hat{\beta_1}$

In general, $\sigma^2$ is not known, but can be estimated from the data. The estimate of $\sigma$ is known as the __________, and is given by the formula

Example 2:

From Advertising data it is found that:

The average cost for TV advertising is 147.0425
The standard deviation of cost for TV advertising is 85.8542363
The average sales is 14.0225
The standard deviation of sales 5.2174566
Correlation coefficient between the cost for TV advertising and sales is 0.7822244

Find the LS estimates for model coefficients ($\hat{\beta_0}$, $\hat{\beta_1}$) when regressing sales onto TV.

Use the lm() function to find the LS estimates for model coefficients ($\hat{\beta_0}$, $\hat{\beta_1}$) when regressing sales onto TV. Compare your answers with a)

State your final simple linear regression model

Interpreating regression coefficients

$\hat{\beta_1}:$ The average increase/decrease in $Y$ for every one unit increase in $X$.
$\hat{\beta_0}:$ The average value of $Y$ when the value of $X = 0$. In most cases, we will find no meaning in $\hat{\beta_0}$.

Example 3: In the advertising data, the sales are recorded in thousands of units and the advertising costs are recorded in thousands of dollars. Interpret the model coefficient estimates you obtained in Example 2.

Confidence intervals for model coefficients $\beta_1$ and $\beta_0$

A $100(1-\alpha)\%$ (example: 95%) confidence interval is defined as a range of values such that with $100(1-\alpha)\%$ (example: 95%) probability, the range will contain the true unknown value of the parameter.

For linear regression, the 95% confidence interval for $\beta_1$ takes the form

That is, there is approximately a 95% chance that the interval

will contain the true value of β1.

Here are some common values for $\beta_1$.

$100(1-α)\%$	$90\%$	$95\%$	$99\%$
$z^∗$	1.645	1.96	2.576

Similarly, a confidence interval for $\beta_0$ approximately takes the form

Example 4:

Table below provides details of the least squares model for the regression of number of units sold on TV advertising budget for the Advertising data.

	Coefficient	Std. error	t-statistic	p-value
`Intercept`	7.0325	0.4578	15.36	< 0.0001
`TV`	0.0475	0.0027	17.67	< 0.0001

Find a 90% confidence interval for $\beta_0$
Find a 90% confidence interval for $\beta_1$

Example 5: Use R to find

Find a 90% confidence interval for $\beta_0$
Find a 90% confidence interval for $\beta_1$

when regressing Sales onto TV in Advertising data.

Hypothesis testing for $\beta_0$ and $\beta_1$

The most common hypothesis test involves testing the null test hypothesis

\[H_0: \text{}\] \[H_a: \text{}\]

Mathematically, this corresponds to testing

\[H_0: \] \[H_a: \] (since if _______ then the model $Y = \beta_0 + \beta_1X + \epsilon$ reduces to ________, and $X$ is not associated with $Y$.)

We usually use four steps to conduct a hypothesis test:

State the null and alternative hypothesis:

Calculate the test statistic: (get/ calculate this value from the $R$ output)

This test statistic has a $t$-distribution with $n−2$ degrees of freedom. The $t$-distribution has a bell shape and for values of $n$ greater than approximately 30 it is quite similar to the normal distribution.

Find the $p$-value:(get this value from the $R$ output)

Make the decision:

If the $p$-value < the given cutoff ($\alpha$) level, we reject $H_0$. Then we say that: there is enough evidence to conclude that there is a linear relationship to exist between $X$ and $Y$.
If the $p$-value > the given cutoff ($\alpha$) level, we do not reject $H_0$. Then we say that: there is not enough evidence to conclude that there is a linear relationship to exist between $X$ and $Y$.

Example 6:

Table below provides details of the least squares model for the regression of number of units sold on TV advertising budget for the Advertising data.

	Coefficient	Std. error	t-statistic	p-value
`Intercept`	7.0325	0.4578	15.36	< 0.0001
`TV`	0.0475	0.0027	17.67	< 0.0001

Perform a complete four step hypothesis test to check whether there is a linear relationship to exist between TV advertising budget and Sales.

State the null and alternative hypothesis:

Calculate the test statistic: (get/ calculate this value from the $R$ output)

Find the $p$-value:(get this value from the $R$ output)

Make the decision:

Example 7:

Use $R$ to perform a complete four step hypothesis test to check whether there is a linear relationship to exist between TV advertising budget and Sales.

Assessing the Accuracy of the Model

The quality of a linear regression fit is typically assessed using two related quantities:

1. Residual Standard Error (RSE)

The RSE is considered a measure of the lack of fit of the model to the data.

$$RSE = $$

If the predictions obtained using the model are very close to the true outcome values—that is, if _________ then RSE will be _________, and we can conclude that the model fits the data very well.

On the other hand, if ___ is very far from ____ for one or more observations, then the RSE may be _________, indicating that the model doesn’t fit the data well.

Units of RSE is same as the unites of the $y$ variable.

Example 8: In the case of the advertising data, find the RSE using R. Interpret this value.

Actual _____ in deviate from the true regression line by approximately _____ units, on average.

2. $R^2$ Statistic

The $R^2$ statistic provides an alternative measure of fit.

\[R^2 = \] where $TSS = \sum(y_i − \bar{y})^2$ is the total sum of squares.

TSS measures the total variance in the response $Y$, and can be thought of as the amount of variability inherent in the response before the regression is performed.

Note that: $0 \leq R^2 \leq 1$
How to interpret $R^2$: $R^2$ measures the proportion of variability in $Y$ that can be explained using $X$.
An $R^2$ statistic that is close to 1 indicates that a large proportion of the variability in the response has been explained by the regression.
An $R^2$ statistic near 0 indicates that the regression did not explain much of the variability in the response
In the the simple linear regression setting, $R^2 = r^2$. Here $r^2$ is the sample correlation.

Example 9: In the case of the advertising data, find the $R^2$ using R. Interpret this value.

Using R, verify that $R^2 = r^2$.

Multiple Linear Regression (MLR)

library(plotly)

p <- plot_ly(data = AdvertisingData, z = ~Sales, x = ~TV, y = ~Radio, opacity = 0.6, color = AdvertisingData$Sales) %>%
  add_markers() 
p

Now we have $p$ distinct predictors (not only one as in simple linear regression). Then the multiple linear regression model takes the form

$$$$

where $X_j$ represents the $j$th predictor and $\beta_j$ quantifies the association between that variable and the response.

How to interpret the $\beta_j$ values: We interpret $\beta_j$ as the average effect on $Y$ for a one unit increase in $X_j$ , holding all other predictors fixed.

Example 10: In the case of the advertising data, write the MLR model using all the available predictors (TV, Radio, Newspaper) to predict the response Sales.

Estimating the Regression Coefficients

As was the case in the simple linear regression setting, the regression coefficients _________ in the MLR model are unknown, and must be estimated. Given estimates _________ we can make predictions using the formula

Example 11: In the case of the advertising data;

find the MLR model using all the available predictors to predict the response Sales in R.
Write the MLR equation
Interpret each coefficient

Correlation matrix

A correlation matrix is a table of correlation coefficients for a set of variables used to determine if a relationship exists between the variables. The coefficient indicates both the strength of the relationship as well as the direction (positive vs. negative correlations)

Some Important Questions:

When we perform multiple linear regression, we usually are interested in answering a few important questions.

Is at least one of the predictors $X_1,X_2, . . . , X_p$ useful in predicting the response?
Do all the predictors help to explain $Y$ , or is only a subset of the predictors useful?
How well does the model fit the data?
Given a set of predictor values, what response value should we predict, and how accurate is our prediction?

We now address each of these questions in turn.

One: Is There a Relationship Between the Response and Predictors?

Recall that in the simple linear regression setting, in order to determine whether there is a relationship between the response and the predictor we can simply check whether ________.

In the multiple regression setting with $p$ predictors, we need to ask whether all of the regression coefficients are zero, i.e. whether _______________.

As in the simple linear regression setting, we use a hypothesis test to answer this question. We test the null hypothesis,

This hypothesis test is performed by computing the F-statistic,

In our class, we will use R or a given R output to get the F-statistic.

Example 11: Conduct a hypothesis test to check if there is a relationship between Sales and the predictor variables Radio, TV and Newspaper. Use the MLR obtained by regressing Sales onto Radio, TV and Newspaper.

Two: Deciding on Important Variables

The task of determining which predictors are associated with the response, in order to fit a single model involving only those predictors, is referred to as ________. The variable selection problem is studied extensively in Chapter 6.

Three: Model Fit

Use $R^2$
Use $RSE = $

Four: Predictions

Confidence interval for average $Y$:

We can compute a confidence interval in order to determine how close ____ will be to ____. Confidence Interval will provide an interval estimate for the average $Y$.

Example 12: Find a 95% CI for average sales when regressing Sales onto Radio, and TV, given that $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in each city.

Given that interval $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in each city, the 95% confidence interval for average sales is ________.

Prediction interval for $Y$ for a future $X$:

Prediction Interval will provide an interval estimate for the $Y$ given a new observation $X$.

Example 13: Find a 95% PI for sales when regressing Sales onto Radio, and TV, given that $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in a particular city.

Given that $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in that particular city the 95% prediction interval for sales is _________.

Regression models with Qualitative Predictors

We use the Credit data set from the ISLR library to illustrate this idea.

Problem: Suppose that we wish to investigate differences in credit card balance between males and females, ignoring the other variables for the moment.

In this data set we have ___ qualitative variables. Namely, __________.

To answer the problem above, we use the variable Gender to predict the Balance.

If a qualitative predictor (also known as a factor) only has __________ levels, then incorporating it into a regression model is very simple.

We simply create an __________ or __________ variable that takes on two possible numerical values.

For example, based on the gender variable, we can create variable a new variable that takes the form

\[\begin{equation} x_i = \begin{cases} 1 & \text{if ith person is female} \\ 0 & \text{if ith person is male} \end{cases} \end{equation}\]

and use this variable as a predictor in the regression equation. This results in the model

\[\begin{equation} y_i = \beta_0 + \beta_1x_i + \epsilon_i = \begin{cases} \beta_0 + \beta_1+ \epsilon_i & \text{if ith person is female} \\ \beta_0 + \epsilon_i & \text{if ith person is male} \end{cases} \end{equation}\]

Use the output above and write the model

\[\begin{equation} \hat{Blanace} = 509.80 + 19.73 \times (Female) = \begin{cases} 509.80 + 19.73 & \text{if person is female} \\ 509.80 & \text{if person is male} \end{cases} \end{equation}\]

How to interpret the coefficients of dummy variables?

509.80: The average credit card debt for males is estimated to be $509.80.

19.73: The average difference in credit card balance between females and males is $19.73.

Example 14: Suppose that we wish to investigate differences in credit card balance using marital status, ignoring the other variables for the moment. Fit a model and interpret its coefficients.

model:

Interpretation:

Example 15: Suppose that we wish to investigate differences in credit card balance using ethnicity, ignoring the other variables for the moment. Fit a model and interpret its coefficients.

What is the baseline category for the Ethnicity variable in this model?
Write the equation of the model

Interpret the coefficients

Interpret the predicted values

Regression models with interactions

Consider the standard linear regression model with two variables,

According to this model, if we increase $X_1$ by one unit, then Y will increase by an average of _____ units. Notice that the presence of $X_2$ does not alter this statement—that is, regardless of the value of $X_2$, a one-unit increase in $X_1$ will lead to a -unit increase in Y . One way of extending this model to allow for ___ effects is to include a third predictor, called an interaction term, which is constructed by computing the product of $X_1$ and $X_2$. This results in the model

Exercise: Consider the Advertising example. write a linear model that uses radio, TV, and an interaction between the two to predict sales.

We can interpret ____ as the increase in the effectiveness of TV advertising for a one unit increase in radio advertising (or vice-versa).

Example 16:

Fit a linear regression model to Advertising dataset with an interaction term that uses radio, TV, and an interaction between the two to predict sales.

Write down your final model.

Perform a hypothesis test to check whether the interaction effect term is useful in the model.

Note: In this example, the p-values associated with TV, radio, and the interaction term all are statistically ________, and so it is obvious that all three variables should be included in the model.

However, it is sometimes the case that an interaction term has a very ______ p-value, but the associated main effects (in this case, TV and radio) do not. The hierarchical principle states that *if we include an interaction in a model, we should also include the main effects, even if the p-values associated with their coefficients are ___ significant.

In other words, if the interaction between $X_1$ and $X_2$ seems important, then we should include both $X_1$ and $X_2$ in the model even if their coefficient estimates have large p-values.

Chapter 3 - Linear Regression